1. Field of the Invention
The present invention relates generally to a semantic interface for a computer system, and more particularly, to a system and method of providing a semantic interface that allows a user to access via a set of user defined words a plurality of services, including command, navigation and substitution, within all contexts of his/her computer system.
2. Related Art
Computers have revolutionized the way individuals in all aspects of life perform tasks. A user interface provides a mechanism for individuals to access all the features and functionalities of their computer. Without a user friendly interface, these features and functions are typically inaccessible to the computer operator. The prevalent user interface in the industry today uses windows, icons, menus and pointing devices. The text stream entered by the user, however, has been essentially ignored.
This window-based user interface (also referred to as a graphical user interface or GUI) was first conceived by Xerox, commercialized by Apple Computers (e.g., the Macintosh), and brought to the mainstream by Microsoft Corporation (e.g., Windows 95). The GUI is powerful for organizing the capabilities and resources available in a computer. It enables the user to incrementally explore and discover his computer""s capabilities and controls. It keeps everything in a convenient visual context, using helpful metaphors, like desktops and windows.
The GUI provides a menu hierarchy which is accessible via a pointing device or mouse. One of the cornerstones of this interface is the ability for the user to interact directly with objects and elements. This can be a great advantage in some cases, but it can result in making simple tasks that are often repeated into tedious choirs of navigation through a maze of GUI windows. To provide the ability to directly manipulate the elements or objects, you must enable the user to work at an xe2x80x9catomicxe2x80x9d level, losing the ability to group a related series of basic actions into one high level action or use of conditionals.
Under this paradigm, we can no longer access or work with objects that are not visible or unknown to us. This situation is not unlike going to eat in another country where we do not know the language. We are thus forced to go to the kitchen and point to whatever food we want. It is clear that we do not want to go to the kitchen in order to eat; we would rather express ourselves using all the richness that our natural language allows. In order to accomplish this goal, the computer has to respond to human language and not the other way around. In reality, we want to be able to go to our favorite restaurant and say xe2x80x9cgive me my usual orderxe2x80x9d and receive exactly what we ordered. This is personal attention and awareness of your eating profile. This is, in essence, what we want from our personal computer.
Advocates of the window-based user interface firmly believe that the user should always be in control. The window-based user interface provides permanent feedback to the user by providing windows with menus. The down side of this is that the user must always be in control even if he does not want to be or he cannot be because of the complexity of the task. Windows 95, for example, has started a trend in allowing the user to delegate his control to an agent, through a concept of smart and thoughtful xe2x80x9cagents.xe2x80x9d Such tasks like un-installing software, are, for example, automatically performed by Windows 95.
There are many tasks that a user must repeat again and again when using a GUI, such as opening certain files and activating certain controls. For such tasks, the GUI presents the user with a single logic set, implemented within the limited screen real estate of his computer monitor. Also, the GUI recognizes none of the user""s words. Even the simplest functions requires the user to change mode from the keyboard to hand/eye mouse control. To use the GUI, he must lift his hand from the keyboard to the mouse. He must also lift his eyes to the screen to locate the desired graphical element, and then manipulate the mouse while visually monitoring the result. This is like having to look at someone every time you want to say a word to that person.
Computer systems must provide users with a mechanism to undo a previous action when a mistake has been made. These same systems must also provide a strong warning if an intended action will be irreversible. This is not a great concern when you are a novice user, but when you become a more experienced user, this feature will turn against you and unnecessarily increase your workload. Take for instance the simple task of copying a file to a floppy that does not have enough free space. The Macintosh window-based interface provides a warning that you must throw away X Kb in order to make space. The user, in an attempt to make space, discards X Kb of data from the floppy and places it in the trash. The user once again attempts to copy the file onto the floppy, only to be told that there is not enough room, but would you like to throw out the trash. This is what you wanted from the start! What was a good feature quickly becomes a nuisance.
The problem stems from the inability of the computer to fully understand even our simplest intentions; it lacks our personality profile. In order to overcome this problem, the computer needs to build a deeper model of each user""s intentions and history in order to better serve the user""s needs and to eliminate unnecessarily repetitive activities. The core requirement here is to provide mechanisms to ascertain how users work and to track their activities in an unobtrusive way.
Current wisdom states that the more static and unchanging our environments, the simpler and better it is for us. As we grow in knowledge and understanding of what the computer can do for us, we are willing to accept changes and learn to cope with them in our quest to increase our personal productivity. Unfortunately, current computer user interfaces have limited abilities to allow a user to express themselves. If computers could communicate with a richer language, it would not be so important that everything have a uniform look and feel.
The computer interface should allow the user to perform any task at any time, irrespective of the application that are currently running. In other words, if a user is working on a word processor and needs to make some calculations, he should not be required to leave his work and open another application via a menu driven user interface to complete the necessary arithmetic operations.
Every computer user has a unique pattern of use. Typically, 80% of a user""s work product is accomplished through repeated use of only 20% of his software""s available features. This is commonly referred to as Pareto""s Rule, and the 20% of the tasks are often referred to as the xe2x80x9cvital few.xe2x80x9d The 80% of available software features and functions that are not needed or used by any particular user must still be available to all other users through the GUI system of menus and windows. Every user""s xe2x80x9c80/20xe2x80x9d profile is unique. Nevertheless, it is the need to organize 100% of the available functionality that necessitates the depth, nesting and complexity of current GUI system. As a result, the GUI is an inefficient fit, to a greater or lesser degree, for every individual user.
Over the years, a number of approaches have been invented to tackle this problem of inefficient fit. Because of their inherent limitations, none have been successful enough to reach the mainstream user. Software entrepreneurs have developed xe2x80x9cshortcutxe2x80x9d utilities of various designs. While not specifically marketed as such, the intention of these utilities is to address each user""s xe2x80x9c80/20xe2x80x9d pattern of often repeated tasks. These xe2x80x9cshortcutxe2x80x9d utilities take two forms: macros triggered by key combinations and icon palette macros.
Macros triggered by key combinations typically take one or both of two forms, macro utilities and text replacement utilities. Macro utility programs provide shortcuts to functions and processes such as opening applications and files, making menu selections, and performing multi-step operations. Macro utilities, such as Tempo, MacroMagic, and Keboard Express for the xe2x80x9cWinTelxe2x80x9d platform and QuicKeys for the Macintosh are all activated by the user via keystroke combinations. Microsoft""s Windows interface offers many key combination shortcuts macros to operate various controls, menus and etc. To activate these macros, the user must press at least one xe2x80x9ccontrolxe2x80x9d key (e.g.,  less than alt greater than ), combined with pressing a single xe2x80x9cnon-controlxe2x80x9d character (e.g.,  less than x greater than ). Users find it very difficult to develop a mnemonically consistent scheme for remembering such key combinations, for two reasons. First, the combinations are mnemonically so arbitrary that it is difficult to use mnemonic logic to memorize the cryptic key combinations. Also, many key combinations only work a given way in specific application programs, further restricting the combinations that are available. The user""s limited ability to remember and reflexively recall more than a few cryptic key combinations severely limit""s the usability of macro utilities. Many people are so intimidated by the cryptic nature of macros they refuse to even consider their use.
Text substitution utilities provide the ability to replace a short string of typed text with long and/or formatted text. For example, a user may define the code word xe2x80x9cevpxe2x80x9d to trigger the substitution to xe2x80x9cExecutive Vice Presidentxe2x80x9d, or define a short code word like xe2x80x9cnadxe2x80x9d to be replaced by a series of pre-defined text lines (name and address in this case). There are several utility software products available to do that within single applications. Text replacement utilities for single applications are, for example, included with Word 7.0 for Windows 95. Other examples include ShortKeys for Windows and both SpellCatcher and TypeIt4Me on the Macintosh platform. Recognition of the user""s words by these utilities is limited to the purpose of replacing one text string with another. These utilities are writer""s aids only. They do not enable the user to also use words for controlling computer processes and functions.
Icon palette utilities are used to give macros a visual presence and context. The macros are activated via mouse clicks. The Icon pallets are an attempt to use a visual interface to overcome the cryptic and therefore hard-to-remember keystroke interface for macro utilities. Often, macro utility products offer icon palettes as a second, alternative interface for accessing the macros. In this approach, a computer macro (process or function) is assigned to a graphical icon, which is presented on an icon bar on the user""s screen. Examples of such utilities are included in Norton Navigator for Window 95 and in both QuickKeys and OneClick on the Macintosh platform.
By definition, these Icon pallet utilities are an extension of the GUI. Screen size, display resolution, and the user""s preference in allocating scarce screen real estate limit the number of icons it makes sense for the user to display on his screen. Given that the users xe2x80x9cvital fewxe2x80x9d can involve scores or hundreds of items, the Icon approach is severely limited by the visual real estate available and the amount of visual complexity the user can tolerate. Moreover, the user must memorize the relationship between the graphic depiction of each icon and the function or process each executes. As the users icon pallet population increases, the distinctiveness of each icon is reduced.
The existing shortcut utilities do not offer the user an integrated approach to creating, managing and using shortcuts for content services, retrieval services and command. Their interfaces are inconsistent and far too difficult to organize and remember. Because the user must assemble his shortcuts using a collection of different software products, he loses a lot of his gains in dealing with cumbersome and time-consuming management of his shortcuts.
It is clear from the above, that the current trend to rely solely on window-based user interfaces has seriously constrained a user""s ability to fully utilize their computer. Although the window-based user interface has revolutionized the computer system, and has allowed millions of people to use computers, we have reached a point where a user""s ability to fully appreciate and utilize all of the features and functionalities of their computer system has been compromised. Thus, what is needed is a system and method that provides a user with an efficient, convenient and natural way to utilize his everyday language to work with applications, files, control commands, and the like, that form his/her xe2x80x9cvital few.xe2x80x9d
The present invention allows a user to use their everyday language competency or user defined code words to operate a computer in a highly efficient way. In short, every word, letter, control character and symbol is actionable. The present invention is based on Pareto""s law, which applies to how people work. Pareto""s law states that people use 20% of all available tools and functionality to accomplish 80% of their tasks. Similarly, 80% of people""s work is accomplished by repeating 20%, or the vital few, of their tasks. By focusing on those activities that enable us to produce most of our work and making them available through simple, natural language-based commands, the present invention enhances a computer user""s productivity dramatically. The present invention provides a more intuitive interface that enhances the operation of the current standard graphical user interface (GUI) in a manner that is simple, richer and natural. By leveraging all of the richness and power inherent in our language, the present invention provides an important tool that allows the personal computer to operate in a manner that is much closer to our natural way of interacting; that is, the way people interact with each other.
The present invention provides a language awareness paradigm, which was born out of a very practical need: to do more with current resources. The basic principles of the language awareness paradigm can be stated very simply:
all commands are natural language-based and/or user-defined.
the basic set of commands are designed to allow users to gain access to their vital few (e.g., applications, documents, controls and functions), which defines each user""s xe2x80x9csweet spotxe2x80x9d of activity, using a least effort path.
all operations and functionality are unobtrusive.
all user""s input is recorded in a context rich format for future reference.
the combined set of all user word preferences, defined commands, and the order in which the commands are stored in memory constitutes a personality profile and are transportable from one computer to another.
Based on the above principles, the present invention provides a user environment, referred to as a semantic user interface (SUI), that compliments the GUI. Via the SUI, the user is enabled to enter action words and interact with the system to control the operations of the computer. The SUI is always monitoring the user""s input text stream in the background.
The SUI thus makes the computer responsive, on a system-wide basis, to the user""s every word. Accordingly, the SUI allows a user to enter action words from any context (i.e., any application or operating system workspace). Action words are a new category of words introduced by the present invention. Action words are thus words that users place into the text stream as requests for specific services from the present invention. There are two types of action words: code words and dual words. Code words are action words the user makes up or which are not part of his natural language lexicon (e.g., not in the standard dictionary). For example, typing xe2x80x9cmswordxe2x80x9d to launch Microsoft""s Word application is an example of entering an code word. Dual words are utterances that can be either ordinary content words or action words, depending on the user""s intention in typing the word. The user may type xe2x80x9cexcelxe2x80x9d because he intends it to be a content word in his application text, or, alternatively, he may type it because he wants to use it as an action word for opening Microsoft Excel. An xe2x80x9caction wordxe2x80x9d can be either a single word or a phrase that includes two or more words.
The action words are then checked against the contents of a wordbase. The wordbase includes a plurality of item records. Each item record includes an action word (i.e., code word and/or dual word) and an associated service script. The service script may perform a content, retrieval, navigation or command service, or a combination of these. If the action word entered by the user is located within the wordbase, the service script associated therewith is executed. Otherwise, the utterance entered by the user is a content word and is ignored by the present invention.
Action words allow a user to launch applications, navigate within applications and control application functions by using their natural language rather than dragging and clicking with a pointing device such as a mouse. The language used is personalized for each user. That is, the action words can be user defined, thus allowing a user to utilize his own lexicon of words to control his/her computer. The present invention allows the user to identify a variety of repetitive tasks and trigger them via their predefined action words. It also enables new types of computer access, information retrieval, and other services to be performed. The present invention works with, and independently of, any software application (e.g., word processor, spreadsheet, presentation package, Internet navigator, and the like). It is thus a context-free semantic user interface, software tool and an application environment.
The present invention saves all information that is entered by the user, and stores this information in a maintenance free environment, referred to as an ActiveWords archive. The present invention records and archives the user""s input text on the fly from whatever application he is working in at that time. The present invention further creates a so called 7xc3x977 data repository, which is a database that is divided into seven categories, each category having seven subcategories. The 7xc3x977 categorization allows a user to record notes, expenses, to do lists, and the like. Finally, the present invention is completely portable. It goes wherever the user goes simply by providing for the user""s personal profile to be downloadable from one computer to any other computer that has the present invention installed.
The user can create a user profile to match his unique language personality. The present invention keeps an archive record of the user""s language preferences, word frequencies, and his utterance behavior. It provides the user with tools for using that archive, in combination with his user profile, to refine his SUI and tailor it to match his habit""s and preferences. Using the SUI thus becomes reflexive, like the use of a mouse becomes reflexive, because it is so easy to learn and operate, and because it operates the same way in all contexts. Finally, the SUI establishes a platform others can use to develop and sell application products that leverage the SUI. By linking the SUI via software agents, any software product can become language aware.
The foregoing and other features and advantages of the invention will be apparent from the following, more particular description of a preferred embodiment of the invention.